Method, system and device for multi-label object detection based on an object detection network

ABSTRACT

A multi-label object detection method based on an object detection network includes: selecting an image of an object to be detected as an input image; based on a trained object detection network, obtaining a class of the object to be detected, coordinates of a center of the object to be detected, and a length and a width of a detection rectangular box according to the input image; and outputting the class of the object to be detected, the coordinates of the center of the object to be detected, and the length and the width of the detection rectangular box. The method of the present invention can perform real-time and accurate object detection on different classes of objects with improved detection speed and accuracy, and can solve the problem of object overlapping and occlusion during the object detection.

CROSS REFERENCE TO THE RELATED APPLICATIONS

This application is the national phase entry of InternationalApplication No. PCT/CN2019/124287, filed on Dec. 10, 2019, which isbased upon and claims priority to Chinese Patent Application No.201910172679.5, filed on Mar. 7, 2019, the entire contents of which areincorporated herein by reference.

TECHNICAL FIELD

The present invention pertains to the field of computer vision, and moreparticularly, relates to a method, system and device for multi-labelobject detection based on an object detection network.

BACKGROUND

With the rapid expansion and maturity of artificial intelligence andmachine learning technologies in recent years, the autonomous objectdetection technologies have become an active area in deep learningresearch in China and other countries and has broad applicationprospects in the fields such as intelligent transportation, medicalimage processing, robotics, and agricultural Internet of things. In reallife, there are a great variety of individual objects with complicateddistributions, and humans can sense the shape, color, texture anddistance of an object to accurately identify the class and position ofthe object. This process, however, has been a challenging task incomputer vision. Therefore, current object detection technologies arefacing the challenge of enabling computers to possess the objectdetection capability close to or even superior to that of humans.

The method in the prior art for researching the object detection mainlyincludes: extracting the color, shape, texture and other features of theobject to obtain an object description, then employing a specificmachine learning algorithm to learn the class, position and contour ofthe object, and finally classifying and locating the object anddelineating the boundaries of the object. However, in some cases, theextracted features are generally difficult to reflect inter-clustervariation and intra-cluster similarity due to the large intra-clustervariation high inter-cluster similarity. Especially when the object hasmultiple labels, the features used by different labels are generally notassociated due to the limitation of traditional feature extractionmethods, which causes much more difficulty in identifying the object. Inthe practical object detection, the presence of object overlapping andocclusion blurs the inter-cluster classification boundary and theintra-cluster distinction, which also poses a great challenge inpractical object detection.

Currently, two main methods for realizing multi-label object detectioninclude a traditional image processing-based method and a deep learningmethod. The traditional image processing-based method for objectdetection includes: first extracting a number of Haar features,scale-invariant feature transform (SIFT) features, histograms oforiented gradient (HOG) features, speeded up robust features (SURF),deformable part model (DPM) features and others from the object image,then using traditional classifiers to obtain the prediction results ofclass and the position information of the object. This method has arelatively fast detection speed but a relatively low detection accuracy,and the features are generally extracted manually, resulting in a largeworkload and a lack of universal applicability. The existing deeplearning method for object detection mainly includes the following twoapproaches. The first approach typically employs a candidateregion-based object detection network such as region-based convolutionalneural network (R-CNN), which has significantly improved accuracy andspeed of detection, but it is difficult to achieve real-time detectionon high-resolution image. The second approach typically employs anobject detection network such as You Only Look Once (YOLO) network thatconverts the object detection problem into a regression problem, whichhas a fast detection speed, but a slightly lower detection accuracy ofalgorithm.

In general, although numerous object detection networks have beendeveloped in the current field, desired results cannot be obtained dueto difficulties in ensuring both the high detection speed and highaccuracy as needed.

SUMMARY

In order to solve the above-mentioned problems in the prior art, thatis, to solve the problem that existing object detection technologiescannot give consideration to both the detection speed and the detectionaccuracy, the present invention provides a multi-label object detectionmethod based on an object detection network, including:

step S10, selecting an image of an object to be detected as an inputimage;

step S20, based on a trained object detection network, obtaining a classof the object to be detected, coordinates of a center of the object tobe detected, and a length and a width of a detection rectangular boxaccording to the input image; and

step S30, outputting the class of the object to be detected, thecoordinates of the center of the object to be detected, and the lengthand the width of the detection rectangular box.

The object detection network is obtained by replacing a low-resolutionfeature layer in a YOLO-V3 network with a densely connectedconvolutional network.

In some preferred embodiments, steps of training the object detectionnetwork include:

step B10, adjusting an attribute of each image in an obtained trainingimage set according to a standard format to obtain a standardizedtraining image set;

step B20, detecting a batch of images in the standardized training imageset by using the object detection network, and calculating a trainingerror of each classifier of the object detection network;

step B30, when a preset number of training iterations is not reached orthe training error is not lower than a preset threshold, obtaining avariation of a parameter of each layer and updating a parameter of theobject detection network by an error back propagation method; and

step B40, detecting the standardized training image set in a batchingsequence by the updated object detection network, and iterativelyupdating the parameter of the object detection network by the error backpropagation method in step B30 until the preset number of the trainingiterations is reached or the training error is lower than the presetthreshold to obtain the trained object detection network.

In some preferred embodiments, the training error is calculated by thefollowing formula:Loss=Error_(coord)+Error_(iou)+Error_(cls)

where, Loss denotes the training error, Error_(coord) denotes aprediction error of the coordinates, Error_(iou) denotes an Intersectionover Union (IoU) error between a predicted bounding box and a truebounding box, and Error_(cls) denotes a classification error.

In some preferred embodiments, the prediction error of the coordinatesis calculated by the following formula:

${Error}_{coord} = {{\lambda_{coord}{\sum\limits_{i = 1}^{S^{2}}{\sum\limits_{j = 0}^{B}{l_{ij}^{obj}\left\lbrack {\left( {x_{i} - {\hat{x}}_{i}} \right)^{2} + \left( {y_{i} - {\hat{y}}_{i}} \right)^{2}} \right\rbrack}}}} + {\lambda_{coord}{\sum\limits_{i = 1}^{S^{2}}{\sum\limits_{j = 0}^{B}{l_{ij}^{obj}\left\lbrack {\left( {\sqrt{w_{i}} - \sqrt{{\hat{w}}_{i}}} \right)^{2} + \left( {\sqrt{h_{i}} - \sqrt{{\hat{h}}_{i}}} \right)^{2}} \right\rbrack}}}}}$

where, λ_(corrd) denotes a weight assigned to the error of thecoordinates; S² denotes the number of grids divided from the inputimage, B denotes the number of bounding boxes generated from each grid;l_(ij) ^(obj)=1 indicates that the object falls into the j^(th) boundingbox of the i^(th) grid, and l_(ij) ^(obj)=0 indicates that the objectdoes not fall into the j^(th) bounding box of the i^(th) grid;{circumflex over (x)}_(i) and ŷ_(i) denote a predicted x coordinatevalue of the center of the bounding box and a predicted y coordinatevalue of the center of the bounding box, respectively; ŵ_(i) and ĥ_(i)denote a predicted length of the bounding box and a predicted width ofthe bounding box, respectively; x_(i) and y_(i) denote a true xcoordinate value of the center of the bounding box and a true ycoordinate value of the center of the bounding box, respectively; andw_(i) and h_(i) denote a true width of the bounding box and a truelength of the bounding box, respectively.

In some preferred embodiments, the IoU error between the predictedbounding box and the true bounding box is calculated by the followingformula:

${Error}_{iou} = {{\sum\limits_{i = 1}^{S^{2}}{\sum\limits_{j = 0}^{B}{l_{ij}^{obj}\left( {C_{i} - {\hat{C}}_{i}} \right)}^{2}}} + {\lambda_{noobj}{\sum\limits_{i = 1}^{S^{2}}{\sum\limits_{j = 0}^{B}{l_{ij}^{obj}\left( {C_{i} - {\hat{C}}_{i}} \right)}^{2}}}}}$

where, λ_(noobj) denotes a weight assigned to the IoU error between thepredicted bounding box and the true bounding box, S² denotes the numberof the grids divided from the input image, B denotes the number of thebounding boxes generated from each grid, l_(ij) ^(obj)=1 indicates thatthe object falls into the j^(th) bounding box of the i^(th) grid, andl_(ij) ^(obj)=0 indicates that the object does not fall into the j^(th)bounding box of the i^(th) grid, Ĉ_(i) denotes a predicted confidencecoefficient over predicting the class, and C_(i) denotes a trueconfidence coefficient.

In some preferred embodiments, the classification error is calculated bythe following formula:

${Error}_{cls} = {\sum\limits_{i = 0}^{S^{2}}{l_{ij}^{obj}{\sum\limits_{c \in {classes}}\left( {{p_{i}(c)} - {{\hat{p}}_{i}(c)}} \right)^{2}}}}$

where, S² denotes the number of the grids divided from the input image,l_(ij) ^(obj)=1 indicates that the object falls into the j^(th) boundingbox of the i^(th) grid, and l_(ij) ^(obj)=1 indicates that the objectdoes not fall into the j^(th) bounding box of the i^(th) grid,{circumflex over (p)}_(i)(c) denotes a predicted probability that theobject in the i^(th) grid belongs to class c, p_(i)(c) denotes a trueprobability that the object in the i^(th) grid belongs to class c.

In some preferred embodiments, before the step B10 of “adjusting anattribute of each image in an obtained training image set according to astandard format”, the method further includes the following step ofperforming enhancement processing on the training image set:

performing data enhancement on each image in the training image set byone or more selected from the group consisting of a color normalizationmethod, a brightness transformation method, an angle transformationmethod, and a blurring processing method, to obtain an enhanced trainingimage set.

According to another aspect of the present invention, a multi-labelobject detection system based on an object detection network isprovided, including an input module, an object detection module, animage enhancement module, an image format standardization module, and anoutput module.

The input module is configured to acquire and input an image of anobject to be detected.

The object detection module is configured to process the image of theobject to be detected to obtain a class of the object to be detected,coordinates of a center of the object to be detected, and a length and awidth of a detection rectangular box.

The image enhancement module is configured to perform data enhancementon a training image set by using one or more selected from the groupconsisting of a color normalization method, a brightness transformationmethod, an angle transformation method, and a blurring processingmethod.

The image format standardization module is configured to adjust anattribute of each image in the training image set according to astandard format.

The output module is configured to output the class of the object to bedetected, the coordinates of the center of the object to be detected,and the length and the width of the detection rectangular box.

According to the third aspect of the present invention, a storage deviceis provided. A plurality of programs are stored in the storage device,and the plurality of programs are loaded and executed by a processor toachieve the multi-label object detection method based on the objectdetection network mentioned above.

According to the fourth aspect of the present invention, a processingdevice is provided, including a processor and a storage device. Theprocessor is configured to execute a plurality of programs. The storagedevice is configured to store the plurality of programs. The pluralityof programs are loaded and executed by the processor to achieve themulti-label object detection method based on the object detectionnetwork mentioned above.

The advantages of the present invention are as follows.

(1) Aiming at the problem of real-time detection of different classes ofobjects in the image, the method of the present invention employs thedensely connected convolutional network (DenseNet) to improve theYOLO-V3 network, thereby realizing real-time and accurate detection ofdifferent classes of objects.

(2) Compared with the faster region-based convolutional neural network(Faster R-CNN) method and other traditional methods, the method of thepresent invention can perform faster and high-accuracy object detectionand can well solve the problem of object overlapping and occlusionduring the object detection.

BRIEF DESCRIPTION OF THE DRAWINGS

Other features, objectives and advantages of the present invention willbe expressly described with reference to the detailed description of thenon-restrictive embodiments and the drawings.

FIG. 1 is a schematic flow chart of the multi-label object detectionmethod based on the object detection network of the present invention;

FIG. 2 is a schematic flow chart showing the optimization process of thetraining image set according to an embodiment of the multi-label objectdetection method based on the object detection network of the presentinvention;

FIG. 3 is a schematic flow chart showing the training process of theobject detection network according to an embodiment of the multi-labelobject detection method based on the object detection network of thepresent invention;

FIG. 4 is a schematic diagram showing the architecture of the objectdetection network according to an embodiment of the multi-label objectdetection method based on the object detection network of the presentinvention; and

FIG. 5 is a schematic flow chart showing the testing process of theobject detection network according to an embodiment of the multi-labelobject detection method based on the object detection network of thepresent invention.

DETAILED DESCRIPTION OF THE EMBODIMENTS

The present invention will be further described in detail hereinafterwith reference to the drawings and embodiments. It can be understoodthat the specific embodiments described herein are only intended toillustrate the present invention, rather than to limit the presentinvention. In addition, for convenience of the description, only partsrelated to the present invention are shown in the drawings.

It should be noted that the embodiments in the present invention andfeatures in the embodiments can be combined without conflicting witheach other. Hereinafter, the present invention will be described indetail with reference to the drawings and embodiments.

Directed to the problems such as low accuracy and low speed of objectdetection using traditional convolutional neural networks, the presentinvention provides a multi-label object detection method based on anobject detection network. According to the present invention, the objectis quickly detected by the regression method and the low-resolutionfeature layer in the object detection network in the prior art isimproved by the DenseNet to achieve feature reuse, thereby improving thefeature utilization rate and detection accuracy.

In the present invention, the multi-label object detection method basedon the object detection network includes the following steps.

Step S10, an image of an object to be detected is selected as an inputimage.

Step S20, based on a trained object detection network, a class of theobject to be detected, coordinates of a center of the object to bedetected, and a length and a width of a detection rectangular box areobtained according to the input image.

Step S30, the class of the object to be detected, the coordinates of thecenter of the object to be detected, and the length and the width of thedetection rectangular box are output.

The object detection network is obtained by replacing a low-resolutionfeature layer in a YOLO-V3 network with a densely connectedconvolutional network.

In order to more clearly illustrate the multi-label object detectionmethod based on the object detection network of the present invention,each step of the method embodiment of the present invention is describedin detail below with reference to FIG. 1.

Specifically, an embodiment of the multi-label object detection methodbased on the object detection network includes the following stepsS10-S30.

Step S10, an image of an object to be detected is selected as an inputimage.

Step S20, based on a trained object detection network, a class of theobject to be detected, coordinates of a center of the object to bedetected, and a length and a width of a detection rectangular boundingbox are obtained according to the input image. The object detectionnetwork is obtained by replacing a low-resolution feature layer in aYOLO-V3 network with a densely connected convolutional network.

As shown in FIG. 3, the first embodiment of the multi-label objectdetection method based on the object detection network includes thefollowing steps.

Step B10, an attribute of each image in an obtained training image setis adjusted according to a standard format to obtain a standardizedtraining image set.

In the present embodiment of the present invention, detection of appleimages at different growth stages in an orchard is taken as an example,and the training image set includes apple images in three stages, i.e.,a young fruit stage, a growth stage, and a mature stage.

In the present invention, formats of the images in the training imageset are standardized into the PASCAL visual object classes (VOC) formatthrough the following steps.

(1) The long side of the image in the image set to be trained isadjusted to 500 pixels, and the short side of the image is adjustedaccording to the scaling of the long side.

(2) The adjusted image is renamed according to the PASCAL VOC format.

(3) Labels are added to the renamed image, bounding boxes are drawn onapples in the image, and the apples in the bounding boxes are classifiedinto three classes including the young fruit stage, the growth stage,and the mature stage that are denoted by 1, 2, and 3, respectively.

(4) The standardized training image set in the PASCAL VOC format isgenerated.

(5) The images in the standardized training image set are adjusted to24n×24n pixel size by a Darknet, where n is a positive integer.

Before the step B10 of “an attribute of each image in an obtainedtraining image set is adjusted according to a standard format”, themulti-label object detection method based on the object detectionnetwork further includes the following step of performing enhancementprocessing on the training image set.

Data enhancement is performed on each image in the training image set byone or more selected from the group consisting of a color normalizationmethod, a brightness transformation method, an angle transformationmethod, and a blurring processing method, to obtain an enhanced trainingimage set.

In order to enhance the detection effect, the image enhancementprocessing can be performed on the images in the training image set.FIG. 2 is a schematic flow chart showing an optimization process of thetraining image set according to an embodiment of the multi-label objectdetection method based on the object detection network of the presentinvention. Image optimization includes image enhancement processing andimage format standardization. The image enhancement processing includesat least one selected from the group consisting of color normalization,a brightness transformation, an angle transformation, and blurringprocessing. The image enhancement processing methods include, but arenot limited to, the color normalization, the brightness transformation,the angle transformation, the blurring processing, and other processingmethods not listed herein.

The processes of the several image enhancement processing methodsincluding the color normalization, the brightness transformation, theangle transformation and the blurring processing are specificallydescribed as follows:

The color normalization method performs the color normalizationprocessing by a color normalization algorithm (i.e., the Gray Worldalgorithm). The Gray World algorithm is based on the gray worldhypothesis, wherein the gray world hypothesis assumes that for an imagewith a number of color changes, the average of the three components red(R), green (G) and blue (B) tends to the same gray value. In a physicalsense, the gray world method assumes that the average reflectance onlight in a natural scene is a fixed value overall, and this fixed valueis approximately “gray”. This hypothesis is forcedly applied to thetraining image set by the color normalization algorithm to eliminate theinfluence of ambient light on the image and obtain the original sceneimage of the training image set.

The brightness transformation method performs the brightnesstransformation processing on the training image set by randomlyselecting different brightness values within a preset brightness range.In the embodiment of the present invention, two values within the rangeof 0.6 to 1.4 times the brightness of the original image are randomlyselected to regulate the brightness of the training image set.

The angle transformation method performs rotations by different anglesand mirroring processing on the training image set. In the embodiment ofthe present invention, rotations by 90°, 180° and 270° and the mirroringprocessing are performed on the original image.

The Blurring processing method performs stochastic blurring processingon the training image set or at least one class of processed imagesafter being subjected to the color normalization, brightnesstransformation, and angle transformation.

480 original images captured in the orchard are processed by the imageenhancement processing methods mentioned above to obtain the enhancedstandardized training image set containing 4,800 images. The 4,800images include 1600 images of apples in each stage of the young fruitstage, the growth stage, and the mature stage, respectively.

Step B20, a batch of images in the standardized training image set aredetected by using the object detection network, and a training error ofeach classifier of the object detection network is calculated.

In the present invention, the down-sampling layers with the resolutionsof 32×32 and 16×16 in the detection network are replaced with theDenseNet to improve the feature utilization rate of the low-resolutionfeature layers in the detection network. FIG. 4 is a schematic diagramshowing the architecture of the object detection network according to anembodiment of the multi-label object detection method based on theobject detection network of the present invention, wherein convolutionallayers denote the convolutional layers, Scale denotes the size of theconvolutional layers, and Convs denotes a convolution operation. TheH_(l) function in the DenseNet layer use a combination function of batchnormalization (BN), rectified linear unit (ReLU), and convolution(Cony), and has the architecture of BN-ReLU-Conv (1×1)-BN-ReLU-Conv(3×3) to realize a nonlinear transformation of the x_(i−1) layer. In thefeature layer with a resolution of 32×32, x_(i) includes 64 sub-featurelayers. H₁ performs a BN-ReLU-Conv (1×1) nonlinear operation on x₀ toobtain a result, and then performs a BN-ReLU-Conv (3×3) operation on theresult. Similarly, H₂ performs an operation same as that mentioned aboveon the feature map formed by [x₀, x₁] to obtain the result x₂, and theobtained result x₂ and [x₀, x₁] are combined into [x₀, x₁, x₂] as theinput of H₃ to obtain the result x₃. Similarly, the obtained result x₃and [x₀, x₁, x₂] are combined into [x₀, x₁, x₂, x₃] as the input of H₄.Finally, the 32×32×512 feature layer combined by [x₀, x₁, x₂, x₃, x₄]continues to propagate forward for training. In the feature layer with aresolution of 16×16, x_(i) includes 128 sub-feature layers, the featurepropagation and feature layer combination same as that mentioned aboveare performed to finally form the 16×16×1024 feature layer thatpropagates forward for training.

The constructed object detection network is initialized. In theembodiment of the present invention, the number of one batch of imagessent to the training model is set to 8. The parameter momentum forfine-tuning the gradient update direction according to the error is setto 0.9. The initial learning rate is set to 0.001. The weight decayregularization term is set to 0.0005. The maximum number of trainingiterations is set to 70000.

The constructed object detection network is trained by the obtainedstandardized training image set, and the images propagate by feedforwardin the object detection network to obtain the training error.Convolution and pooling operations are continuously performed on eachread image until the object detection network outputs the training errorof each classifier.

The training error is calculated by formula (1):Loss=Error_(coord) Error_(iou)+Error_(cls)  formula(1)

where, Loss denotes the training error, Error_(coord) denotes aprediction error of the coordinates, Error_(iou) denotes an IoU errorbetween a predicted bounding box and a true bounding box, andError_(cls) denotes a classification error.

The prediction error of the coordinates is calculated by formula (2):

$\begin{matrix}{{Error}_{coord} = {{\lambda_{coord}{\sum\limits_{i = 1}^{S^{2}}{\sum\limits_{j = 0}^{B}{l_{ij}^{obj}\left\lbrack {\left( {x_{i} - {\hat{x}}_{i}} \right)^{2} + \left( {y_{i} - {\hat{y}}_{i}} \right)^{2}} \right\rbrack}}}} + {\lambda_{coord}{\sum\limits_{i = 1}^{S^{2}}{\sum\limits_{j = 0}^{B}{l_{ij}^{obj}\left\lbrack {\left( {\sqrt{w_{i}} - \sqrt{{\hat{w}}_{i}}} \right)^{2} + \left( {\sqrt{h_{i}} - \sqrt{{\hat{h}}_{i}}} \right)^{2}} \right\rbrack}}}}}} & {{formula}\mspace{14mu}(2)}\end{matrix}$

where, λ_(corrd) denotes a weight assigned to the error of thecoordinates; S² denotes the number of grids divided from the inputimage; B denotes the number of bounding boxes generated from each grid;l_(ij) ^(obj)=1 indicates that the object falls into the j^(th) boundingbox of the i^(th) grid, and l_(ij) ^(obj)=0 indicates that the objectdoes not fall into the j^(th) bounding box of the i^(th) grid;{circumflex over (x)}_(i) and ŷ_(i) denote a predicted x coordinatevalue of the center of the bounding box and a predicted y coordinatevalue of the center of the bounding box, respectively; ŵ_(i) and ĥ_(i)denote a predicted length of the bounding box and a predicted width ofthe bounding box, respectively; x_(i) and y_(i) denote a true xcoordinate value of the center of the bounding box and a true ycoordinate value of the center of the bounding box, respectively; andw_(i) and h_(i) denote a true width of the bounding box and a truelength of the bounding box, respectively.

The IoU error is calculated by formula (3):

$\begin{matrix}{{Error}_{iou} = {{\sum\limits_{i = 1}^{S^{2}}{\sum\limits_{j = 0}^{B}{l_{ij}^{obj}\left( {C_{i} - {\hat{C}}_{i}} \right)}^{2}}} + {\lambda_{noobj}{\sum\limits_{i = 1}^{S^{2}}{\sum\limits_{j = 0}^{B}{l_{ij}^{obj}\left( {C_{i} - {\hat{C}}_{i}} \right)}^{2}}}}}} & {{formula}\mspace{14mu}(3)}\end{matrix}$

where, λ_(noobj) denotes a weight assigned to the IoU error, S² denotesthe number of the grids divided from the input image, B denotes thenumber of the bounding boxes generated from each grid, l_(ij) ^(obj)=1indicates that the object falls into the j^(th) bounding box of thei^(th) grid, and l_(ij) ^(obj)=0 indicates that the object does not fallinto the j^(th) bounding box of the i^(th) grid, Ĉ_(i) denotes apredicted confidence coefficient over predicting the class, and C_(i)denotes a true confidence coefficient.

The classification error is calculated by formula (3):

$\begin{matrix}{{Error}_{cls} = {\sum\limits_{i = 0}^{S^{2}}{l_{ij}^{obj}{\sum\limits_{c \in {classes}}\left( {{p_{i}(c)} - {{\hat{p}}_{i}(c)}} \right)^{2}}}}} & {{formula}\mspace{14mu}(4)}\end{matrix}$

where, S² denotes the number of the grids divided from the input image,l_(ij) ^(obj)=1 indicates that the object falls into the j^(th) boundingbox of the i^(th) grid, and l_(ij) ^(obj)=0 indicates that the objectdoes not fall into the j^(th) bounding box of the i^(th) grid,{circumflex over (p)}_(i) (c) denotes a predicted probability that theobject in the i^(th) grid belongs to class c, p_(i)(c) denotes a trueprobability that the object in the i^(th) grid belongs to class c.

Step B30, when a preset number of training iterations is not reached orthe training error is not lower than a preset threshold, the error backpropagation method is employed to obtain a variation of a parameter ofeach layer and update a parameter of the object detection network.

Step B40, the standardized training image set is detected in a batchingsequence by the updated object detection network, and the parameter ofthe object detection network is iteratively updated by the error backpropagation method in step B30 until the preset number of the trainingiterations is reached or the training error is lower than the presetthreshold to obtain the trained object detection network.

In the embodiment of the present invention, the maximum number oftraining iterations is set to 70,000. When the number of training stepsreaches 70,000 or the calculated training error is lower than the presetthreshold, the training of the object detection network is completed,and the object detection network is employed to detect the class of theobject to be detected.

Step S30, the class of the object to be detected, the coordinates of thecenter of the object to be detected, and the length and the width of thedetection rectangular bounding box are output.

In the embodiment of the present invention, the trained object detectionnetwork is tested and evaluated. FIG. 5 is a schematic flow chartshowing the testing process of the object detection network according toan embodiment of the multi-label object detection method based on theobject detection network of the present invention. Specifically, thetesting process of the object detection network includes the followingsteps:

(1) The above-mentioned image optimization method of the training imageset is employed to perform image format standardization processing andimage enhancement processing on the images of the test image set.

In the embodiment of the present invention, 90 images of apples atdifferent stages are used as the test image set to test the performanceof the optimized object detection network, wherein the 90 images ofapples are directly collected at the orchard and have a resolution of3000×3000.

(2) The YOLO-V2 network, the YOLO-V3 network, the Faster R-CNN, and theoptimized object detection network of the present invention are employedto detect the processed test image set to obtain the object class of thecorresponding image in the test image set and to obtain the precisionrate, the recall rate, the harmonic mean (F1 score) of the precisionrate and the recall rate, the IoU value between the predicted boundingbox and the true bounding box, and the average detection time.

The result thereof is shown in Table 1:

TABLE 1 Faster Method of the Model class YOLO-V2 YOLO-V3 R-CNN presentinvention F1 score 0.738 0.793 0.801 0.817 IoU value 0.805 0.869 0.8730.896 Average detection 0.273 0.296 2.42 0.304 time (s)

As shown in Table 1, in terms of the object detection performance, theF1 score of the trained object detection network of the presentinvention is 0.817, which is 0.016 higher than that of the Faster R-CNNmodel with the highest F1 score among the other three models andindicates that the comprehensive performance of the recall rate andprecision rate of the optimized object detection network of the presentinvention is higher than that of the other three models. The IoU valueof the trained object detection network of the present invention is0.896, which is higher than that of the other three models, andindicates that the detection accuracy of the optimized object detectionnetwork of the present invention is higher than that of the other threemodels. The average detection time of the trained object detectionnetwork of the present invention is 0.304 s, which can achieve real-timedetection of apples in high-resolution images. In practical detection,the detection accuracy of the optimized object detection network of thepresent invention is significantly higher than that of the other threemodels, which reflects the superiority of the method of the presentinvention.

According to the second embodiment of the present invention, amulti-label object detection system based on an object detection networkis provided, including an input module, an object detection module, animage enhancement module, an image format standardization module, and anoutput module.

The input module is configured to acquire and input an image of anobject to be detected.

The object detection module is configured to process the image of theobject to be detected to obtain a class of the object to be detected,coordinates of a center of the object to be detected, and a length and awidth of a detection rectangular box.

The image enhancement module is configured to perform data enhancementon a training image set by using one or more selected from the groupconsisting of a color normalization method, a brightness transformationmethod, an angle transformation method, and a blurring processingmethod.

The image format standardization module is configured to adjust anattribute of each image in the training image set according to astandard format.

The output module is configured to output the class of the object to bedetected, the coordinates of the center of the object to be detected,and the length and the width of the detection rectangular box.

Those skilled in the art can clearly understand that for the convenienceand conciseness of the description, the specific working process andrelated description of the above-mentioned system can refer to thecorresponding process of the method in the foregoing embodiment, whichare not repeatedly described herein.

It should be noted that the multi-label object detection system based onthe object detection network in the above embodiments is onlyexemplified by the division of the above functional modules. Inpractical applications, the above-mentioned functions can be allocatedto different function modules as needed, namely, the modules or steps inthe embodiment of the present invention are further decomposed orcombined. For example, the modules in the above embodiments may becombined into one module, or split into multiple sub-modules to completeall or part of the functions described above. The designations of themodules and steps involved in the embodiments of the present inventionare only used for distinguishing these modules or steps, and should notbe construed as an improper limitation on the present invention.

According to the third embodiment of the present invention, a storagedevice is provided. A plurality of programs are stored in the storagedevice, and the plurality of programs are loaded and executed by aprocessor to achieve the multi-label object detection method based onthe object detection network mentioned above.

According to the fourth embodiment of the present invention, aprocessing device is provided, including a processor and a storagedevice. The processor is configured to execute a plurality of programs.The storage device is configured to store the plurality of programs. Theplurality of programs are loaded and executed by the processor toachieve the multi-label object detection method based on the objectdetection network mentioned above.

Those skilled in the art can clearly understand that for the convenienceand conciseness of the description, the specific working process andrelated description of the storage device and processing devicedescribed above can refer to the corresponding process of the method inthe foregoing embodiments, which are not repeatedly described herein.

Those skilled in the art can realize that the modules, and steps ofmethod described in the embodiments herein can be implemented byelectronic hardware, computer software, or a combination of theelectronic hardware and the computer software. The programscorresponding to modules of software and steps of method can be storedin a random access memory (RAM), a memory, a read-only memory (ROM), anelectrically programmable ROM, an electrically erasable programmableROM, a register, a hard disk, a removable disk, a compact disc-read onlymemory (CD-ROM) or any other form of storage mediums known in thetechnical field. In the above description, the composition and steps ofeach embodiment have been described in terms of the functions to clearlyexplain the interchangeability of electronic hardware and software.Whether these functions are performed by electronic hardware or softwaredepends on the specific application and designed constraint conditionsof the technical solution. Those skilled in the art can use differentmethods to implement the described functions for each specificapplication, but such implementation should not be construed to bebeyond the scope of the present invention.

The terminology “include/comprise” and any other similar terminologiesare used to cover non-exclusive inclusions, so that a process, method,article, equipment or device including a series of elements not onlyinclude these elements, but also include other elements that are notexplicitly listed, or include elements inherent in the process, method,article, equipment or device.

Hereto, the technical solutions of the present invention have beendescribed in combination with the preferred embodiments with referenceto the drawings. However, it is easily understood by those skilled inthe art that the scope of protection of the present invention isobviously not limited to these specific embodiments. Those skilled inthe art can make equivalent modifications or replacements to relatedtechnical features without departing from the principles of the presentinvention, and the technical solutions obtained by these modificationsor replacements shall fall within the scope of protection of the presentinvention.

What is claimed is:
 1. A multi-label object detection method based on anobject detection network, comprising: step S10, selecting an image of anobject to be detected as an input image; step S20, based on the objectdetection network, obtaining a class of the object to be detected,coordinates of a center of the object to be detected, and a length and awidth of a detection rectangular box according to the input image; andstep S30, outputting the class of the object to be detected, thecoordinates of the center of the object to be detected, and the lengthand the width of the detection rectangular box; wherein the objectdetection network is trained and obtained by replacing a low-resolutionfeature layer in a You Only Look Once-V3 (YOLO-V3) network with adensely connected convolutional network.
 2. The multi-label objectdetection method based on the object detection network according toclaim 1, wherein, a method of training the object detection networkcomprises the following steps: step B10, adjusting an attribute of eachimage in an obtained training image set according to a standard formatto obtain a standardized training image set; step B20, detecting a batchof images in the standardized training image set by using the objectdetection network, and calculating a training error of each classifierof the object detection network; step B30, when a preset number oftraining iterations is not reached or the training error is greater thanor equal to a preset threshold, obtaining a variation of a parameter ofeach layer in the object detection network and updating a parameter ofthe object detection network by an error back propagation method; andstep B40, detecting the standardized training image set in a batchingsequence after the parameter of the object detection network is updated,and iteratively updating the parameter of the object detection networkby the error back propagation method in step B30 until the preset numberof the training iterations is reached or the training error is lowerthan the preset threshold to obtain the object detection network.
 3. Themulti-label object detection method based on the object detectionnetwork according to claim 2, wherein, the training error is calculatedby the following formula:Loss=Error_(coord) Error_(iou)+Error_(cls) where, Loss denotes thetraining error, Error_(coord) denotes a prediction error of thecoordinates, Error_(iou) denotes an Intersection over Union (IoU) errorbetween a predicted bounding box and a true bounding box, andError_(cls) denotes a classification error.
 4. The multi-label objectdetection method based on the object detection network according toclaim 3, wherein, the prediction error of the coordinates is calculatedby the following formula:${Error}_{coord} = {{\lambda_{coord}{\sum\limits_{i = 1}^{S^{2}}{\sum\limits_{j = 0}^{B}{l_{ij}^{obj}\left\lbrack {\left( {x_{i} - {\hat{x}}_{i}} \right)^{2} + \left( {y_{i} - {\hat{y}}_{i}} \right)^{2}} \right\rbrack}}}} + {\lambda_{coord}{\sum\limits_{i = 1}^{S^{2}}{\sum\limits_{j = 0}^{B}{l_{ij}^{obj}\left\lbrack {\left( {\sqrt{w_{i}} - \sqrt{{\hat{w}}_{i}}} \right)^{2} + \left( {\sqrt{h_{i}} - \sqrt{{\hat{h}}_{i}}} \right)^{2}} \right\rbrack}}}}}$where, λ_(corrd) denotes a weight assigned to the prediction error ofthe coordinates; S² denotes a number of grids divided from the inputimage; B denotes a number of bounding boxes generated from each grid ofthe grids divided from the input image; l_(ij) ^(obj)=1 indicates thatthe object falls into a j^(th) bounding box of an i^(th) grid of thegrids divided from the input image, and l_(ij) ^(obj)=0 indicates thatthe object does not fall into the j^(th) bounding box of the i^(th)grid; {circumflex over (x)}_(i) and ŷ_(i) denote a predicted xcoordinate value of the center of the bounding box and a predicted ycoordinate value of the center of the bounding box, respectively; ŵ_(i)and ĥ_(i) denote a predicted length of the bounding box and a predictedwidth of the bounding box, respectively; x_(i) and y_(i) denote a true xcoordinate value of the center of the bounding box and a true ycoordinate value of the center of the bounding box, respectively; andw_(i) and h_(i) denote a true length of the bounding box and a truewidth of the bounding box, respectively.
 5. The multi-label objectdetection method based on the object detection network according toclaim 3, wherein, the IoU error between the predicted bounding box andthe true bounding box is calculated by the following formula:${Error}_{iou} = {{\sum\limits_{i = 1}^{S^{2}}{\sum\limits_{j = 0}^{B}{l_{ij}^{obj}\left( {C_{i} - {\hat{C}}_{i}} \right)}^{2}}} + {\lambda_{noobj}{\sum\limits_{i = 1}^{S^{2}}{\sum\limits_{j = 0}^{B}{l_{ij}^{obj}\left( {C_{i} - {\hat{C}}_{i}} \right)}^{2}}}}}$where, λ_(noobj) denotes a weight assigned to the IoU error between thepredicted bounding box and the true bounding box, S² denotes a number ofgrids divided from the input image, B denotes a number of bounding boxesgenerated from each grid of the grids divided from the input image,l_(ij) ^(obj)=1 indicates that the object falls into a j^(th) boundingbox of an i^(th) grid of the grids divided from the input image, andl_(ij) ^(obj)=0 indicates that the object does not fall into the j^(th)bounding box of the i^(th) grid, Ĉ_(i) denotes a predicted confidencecoefficient over predicting the class, and C_(i) denotes a trueconfidence coefficient.
 6. The multi-label object detection method basedon the object detection network according to claim 3, wherein, theclassification error is calculated by the following formula:${Error}_{cls} = {\sum\limits_{i = 0}^{S^{2}}{l_{ij}^{obj}{\sum\limits_{c \in {classes}}\left( {{p_{i}(c)} - {{\hat{p}}_{i}(c)}} \right)^{2}}}}$where, S² denotes a number of a grids divided from the input image,l_(ij) ^(obj)=1 indicates that the object falls into a j^(th) boundingbox of an i^(th) grid, and l_(ij) ^(obj)=0 indicates that the objectdoes not fall into the j^(th) bounding box of the i^(th) grid,{circumflex over (p)}_(i)(c) denotes a predicted probability that theobject in the i^(th) grid belongs to a class c, p_(i)(c) denotes a trueprobability that the object in the i^(th) grid belongs to the class c.7. The multi-label object detection method based on the object detectionnetwork according to claim 2, wherein, before the step B10 of “adjustingthe attribute of the each image in the obtained training image setaccording to the standard format”, the multi-label object detectionmethod further comprises: performing a data enhancement on the eachimage in the obtained training image set to obtain an enhanced trainingimage set, wherein, a method of performing the data enhancementcomprises one or more selected from the group consisting of a colornormalization method, a brightness transformation method, an angletransformation method, and a blurring processing method.
 8. Amulti-label object detection system based on an object detectionnetwork, comprising an input module, an object detection module, animage enhancement module, an image format standardization module, and anoutput module; wherein the input module is configured to acquire andinput an image of an object to be detected; the object detection moduleis configured to obtain a class of the object to be detected,coordinates of a center of the object to be detected, and a length and awidth of a detection rectangular box based on the object detectionnetwork after being trained according to the input image; the imageenhancement module is configured to perform a data enhancement on atraining image set by using one or more selected from the groupconsisting of a color normalization method, a brightness transformationmethod, an angle transformation method, and a blurring processingmethod; the image format standardization module is configured to adjustan attribute of each image in the training image set according to astandard format; and the output module is configured to output the classof the object to be detected, the coordinates of the center of theobject to be detected, and the length and the width of the detectionrectangular box.
 9. A storage device, wherein a plurality of programsare stored in the storage device, and the plurality of programs areloaded and executed by a processor to achieve the multi-label objectdetection method based on the object detection network according toclaim
 1. 10. A processing device, comprising: a processor, wherein theprocessor is configured to execute a plurality of programs, and astorage device, wherein the storage device is configured to store theplurality of programs; wherein the plurality of programs are loaded andexecuted by the processor to achieve the multi-label object detectionmethod based on the object detection network according to claim
 1. 11.The storage device according to claim 9, wherein, a method of trainingthe object detection network comprises the following steps: step B10,adjusting an attribute of each image in an obtained training image setaccording to a standard format to obtain a standardized training imageset; step B20, detecting a batch of images in the standardized trainingimage set by using the object detection network, and calculating atraining error of each classifier of the object detection network; stepB30, when a preset number of training iterations is not reached or thetraining error is greater than or equal to a preset threshold, obtaininga variation of a parameter of each layer in the object detection networkand updating a parameter of the object detection network by an errorback propagation method; and step B40, detecting the standardizedtraining image set in a batching sequence after the parameter of theobject detection network is updated, and iteratively updating theparameter of the object detection network by the error back propagationmethod in step B30 until the preset number of the training iterations isreached or the training error is lower than the preset threshold toobtain the object detection network.
 12. The storage device according toclaim 11, wherein, the training error is calculated by the followingformula:Loss=Error_(coord) Error_(iou)+Error_(cls) where, Loss denotes thetraining error, Error_(coord) denotes a prediction error of thecoordinates, Error_(iou) denotes an Intersection over Union (IoU) errorbetween a predicted bounding box and a true bounding box, andError_(cls) denotes a classification error.
 13. The storage deviceaccording to claim 12, wherein, the prediction error of the coordinatesis calculated by the following formula:${Error}_{coord} = {{\lambda_{coord}{\sum\limits_{i = 1}^{S^{2}}{\sum\limits_{j = 0}^{B}{l_{ij}^{obj}\left\lbrack {\left( {x_{i} - {\hat{x}}_{i}} \right)^{2} + \left( {y_{i} - {\hat{y}}_{i}} \right)^{2}} \right\rbrack}}}} + {\lambda_{coord}{\sum\limits_{i = 1}^{S^{2}}{\sum\limits_{j = 0}^{B}{l_{ij}^{obj}\left\lbrack {\left( {\sqrt{w_{i}} - \sqrt{{\hat{w}}_{i}}} \right)^{2} + \left( {\sqrt{h_{i}} - \sqrt{{\hat{h}}_{i}}} \right)^{2}} \right\rbrack}}}}}$where, λ_(corrd) denotes a weight assigned to the prediction error ofthe coordinates; S² denotes a number of grids divided from the inputimage; B denotes a number of bounding boxes generated from each grid ofthe grids divided from the input image; l_(ij) ^(obj)=1 indicates thatthe object falls into a j^(th) bounding box of an i^(th) grid of thegrids divided from the input image, and l_(ij) ^(obj)=0 indicates thatthe object does not fall into the j^(th) bounding box of the i^(th)grid; {circumflex over (x)}_(i) and ŷ_(i) denote a predicted xcoordinate value of the center of the bounding box and a predicted ycoordinate value of the center of the bounding box, respectively; ŵ_(i)and ĥ_(i) denote a predicted length of the bounding box and a predictedwidth of the bounding box, respectively; x_(i) and y_(i) denote a true xcoordinate value of the center of the bounding box and a true ycoordinate value of the center of the bounding box, respectively; andw_(i) and h_(i) denote a true length of the bounding box and a truewidth of the bounding box, respectively.
 14. The storage deviceaccording to claim 12, wherein, the IoU error between the predictedbounding box and the true bounding box is calculated by the followingformula:${Error}_{iou} = {{\sum\limits_{i = 1}^{S^{2}}{\sum\limits_{j = 0}^{B}{l_{ij}^{obj}\left( {C_{i} - {\hat{C}}_{i}} \right)}^{2}}} + {\lambda_{noobj}{\sum\limits_{i = 1}^{S^{2}}{\sum\limits_{j = 0}^{B}{l_{ij}^{obj}\left( {C_{i} - {\hat{C}}_{i}} \right)}^{2}}}}}$where, λ_(noobj) denotes a weight assigned to the IoU error between thepredicted bounding box and the true bounding box, S² denotes a number ofgrids divided from the input image, B denotes a number of bounding boxesgenerated from each grid of the grids divided from the input image,l_(ij) ^(obj)=1 indicates that the object falls into a j^(th) boundingbox of an i^(th) grid of the grids divided from the input image, andl_(ij) ^(obj)=0 indicates that the object does not fall into the j^(th)bounding box of the i^(th) grid, Ĉ_(i) denotes a predicted confidencecoefficient over predicting the class, and C_(i) denotes a trueconfidence coefficient.
 15. The storage device according to claim 12,wherein, the classification error is calculated by the followingformula:${Error}_{cls} = {\sum\limits_{i = 0}^{S^{2}}{l_{ij}^{obj}{\sum\limits_{c \in {classes}}\left( {{p_{i}(c)} - {{\hat{p}}_{i}(c)}} \right)^{2}}}}$where, S² denotes a number of a grids divided from the input image,l_(ij) ^(obj)=1 indicates that the object falls into a j^(th) boundingbox of an i^(th) grid, and l_(ij) ^(obj)=1 indicates that the objectdoes not fall into the j^(th) bounding box of the i^(th) grid,{circumflex over (p)}_(i) (c) denotes a predicted probability that theobject in the i^(th) grid belongs to a class c, p_(i)(c) denotes a trueprobability that the object in the i^(th) grid belongs to the class c.16. The storage device according to claim 11, wherein, before the stepB10 of “adjusting the attribute of the each image in the obtainedtraining image set according to the standard format”, the multi-labelobject detection method further comprises: performing a data enhancementon the each image in the obtained training image set to obtain anenhanced training image set, wherein, a method of performing the dataenhancement comprises one or more selected from the group consisting ofa color normalization method, a brightness transformation method, anangle transformation method, and a blurring processing method.
 17. Thestorage device according to claim 9, wherein, a method of training theobject detection network comprises the following steps: step B10,adjusting an attribute of each image in an obtained training image setaccording to a standard format to obtain a standardized training imageset; step B20, detecting a batch of images in the standardized trainingimage set by using the object detection network, and calculating atraining error of each classifier of the object detection network; stepB30, when a preset number of training iterations is not reached or thetraining error is greater than or equal to a preset threshold, obtaininga variation of a parameter of each layer in the object detection networkand updating a parameter of the object detection network by an errorback propagation method; and step B40, detecting the standardizedtraining image set in a batching sequence after the parameter of theobject detection network is updated, and iteratively updating theparameter of the object detection network by the error back propagationmethod in step B30 until the preset number of the training iterations isreached or the training error is lower than the preset threshold toobtain the object detection network.
 18. The storage device according toclaim 11, wherein, the training error is calculated by the followingformula:Loss=Error_(coord) Error_(iou)+Error_(cls) where, Loss denotes thetraining error, Error_(coord) denotes a prediction error of thecoordinates, Error_(iou) denotes an Intersection over Union (IoU) errorbetween a predicted bounding box and a true bounding box, andError_(cls) denotes a classification error.
 19. The storage deviceaccording to claim 12, wherein, the prediction error of the coordinatesis calculated by the following formula:${Error}_{coord} = {{\lambda_{coord}{\sum\limits_{i = 1}^{S^{2}}{\sum\limits_{j = 0}^{B}{l_{ij}^{obj}\left\lbrack {\left( {x_{i} - {\hat{x}}_{i}} \right)^{2} + \left( {y_{i} - {\hat{y}}_{i}} \right)^{2}} \right\rbrack}}}} + {\lambda_{coord}{\sum\limits_{i = 1}^{S^{2}}{\sum\limits_{j = 0}^{B}{l_{ij}^{obj}\left\lbrack {\left( {\sqrt{w_{i}} - \sqrt{{\hat{w}}_{i}}} \right)^{2} + \left( {\sqrt{h_{i}} - \sqrt{{\hat{h}}_{i}}} \right)^{2}} \right\rbrack}}}}}$where, λ_(corrd) denotes a weight assigned to the prediction error ofthe coordinates; S² denotes a number of grids divided from the inputimage; B denotes a number of bounding boxes generated from each grid ofthe grids divided from the input image; l_(ij) ^(obj)=1 indicates thatthe object falls into a j^(th) bounding box of an i^(th) grid of thegrids divided from the input image, and l_(ij) ^(obj)=0 indicates thatthe object does not fall into the j^(th) bounding box of the i^(th)grid; {circumflex over (x)}_(i) and ŷ_(i) denote a predicted xcoordinate value of the center of the bounding box and a predicted ycoordinate value of the center of the bounding box, respectively; ŵ_(i)and ĥ_(i) denote a predicted length of the bounding box and a predictedwidth of the bounding box, respectively; x_(i) and y_(i) denote a true xcoordinate value of the center of the bounding box and a true ycoordinate value of the center of the bounding box, respectively; andw_(i) and h_(i) denote a true length of the bounding box and a truewidth of the bounding box, respectively.
 20. The storage deviceaccording to claim 12, wherein, the IoU error between the predictedbounding box and the true bounding box is calculated by the followingformula:${Error}_{iou} = {{\sum\limits_{i = 1}^{S^{2}}{\sum\limits_{j = 0}^{B}{l_{ij}^{obj}\left( {C_{i} - {\hat{C}}_{i}} \right)}^{2}}} + {\lambda_{noobj}{\sum\limits_{i = 1}^{S^{2}}{\sum\limits_{j = 0}^{B}{l_{ij}^{obj}\left( {C_{i} - {\hat{C}}_{i}} \right)}^{2}}}}}$where, λ_(noobj) denotes a weight assigned to the IoU error between thepredicted bounding box and the true bounding box, S² denotes a number ofgrids divided from the input image, B denotes a number of bounding boxesgenerated from each grid of the grids divided from the input image,l_(ij) ^(obj)=1 indicates that the object falls into a j^(th) boundingbox of an i^(th) grid of the grids divided from the input image, andl_(ij) ^(obj)=1 indicates that the object does not fall into the j^(th)bounding box of the i^(th) grid, Ĉ_(i) denotes a predicted confidencecoefficient over predicting the class, and C_(i) denotes a trueconfidence coefficient.